Developers often design applications that are loosely coupled to each other instead of being tightly linked. This configuration is advantageous and beneficial, but introduces the problem of how to gather and distribute information that spans the application. This problem may be further compounded if the line between the applications and the enterprise or consumer is not well defined. Information or data is often required to be collected from many sources or endpoints and distributed to many destinations or sinks. For example, an enterprise application may push data to a social media platform or to mobile devices. The data may come from an information source such as an RSS feed or a Topic Pub/Sub.
Traditionally, the problem of distributing data is solved by developing a “one-off” or custom solution that pumps data between specific information sources and sinks. Generally, the data pump has to poll information sources, which may use a lot of compute resources, and thereby requires a dedicated machine to poll the information sources. This configuration causes additional problems because pull sources may or may not have data available at all times. This causes spikes in the compute requirements when data is available and wasted resources when the pump is simply polling to see if data exists. In some configurations, this is addressed by providing a notification or activation message when data becomes available. These notification mechanisms may be built into information pull sources such as queues.
Historically, data feed systems were limited because the data pump must be close to either the pull source or the push sink and is generally built stand-alone or purpose-built application that is not capable of future connectivity or integration with other systems. Polling from the pull sources can require a lot of compute resources that could be put to better use. Additionally, getting data from a pull source to a push sink invariably requires some kind of transformation of the data that is generally hand coded.